Selecting Near-Optimal Learners via Incremental Data Allocation

We study a novel machine learning (ML) problem setting of sequentially allocating smallsubsets of training data amongst a large set of classifiers. The goal is to select a classifierthat will give near-optimal accuracy when trained on all data, while also minimizing the costof misallocated samples. This is motivated by large modern datasets and ML toolkits withmany combinations of learning algorithms and hyper-parameters. Inspired by the principle of“optimism under uncertainty,” we propose an innovative strategy, Data Allocation using UpperBounds (DAUB), which robustly achieves these objectives across a variety of real-world datasets.We further develop substantial theoretical support for DAUB in an idealized setting wherethe expected accuracy of a classifier trained on n samples can be known exactly. Under theseconditions we establish a rigorous sub-linear bound on the regret of the approach (in termsof misallocated data), as well as a rigorous bound on suboptimality of the selected classifier.Our accuracy estimates using real-world datasets only entail mild violations of the theoreticalscenario, suggesting that the practical behavior of DAUB is likely to approach the idealizedbehavior.